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DETAILED ACTION 

All of the examiner's suggestions presented herein below have been assumed for examination 
purposes, unless otherwise noted. 

Amendments 

1 . This office action is responsive to the preliminary claim amendment received on 
1/25/2005. 

* 

Priority 

2. Acknowledgment is made of applicant's claim for foreign priority based on an application 
filed on 1/25/2005. It is noted, however, that applicant has not filed a certified copy of the 
PCT/JP02/07649 application as required by 35 U.S.C. 1 19(b). 

Drawings 

3. The following is a quote from 37 CFR 1.84(u)(l): 
View numbers must be preceded by the abbreviation "FIG." 

4. The drawings are objected to under 37 CFR 1 .84(u)(l) for failing to properly capitalize 
the view numbers - suggest capitalizing (e.g. "Fig. 1" to "FIG. 1"). 

5. The following is a quote from 37 CFR 1 .84(t): 

(t) Numbering of sheets of drawings. The sheets of drawings should be numbered in consecutive Arabic numerals, 
starting with 1, within the sight as defined in paragraph (g) of this section. These numbers, if present, must be placed in 
the middle of the top of the sheet, but not in the margin. The numbers can be placed on the right-hand side if the 
drawing extends too close to the middle of the top edge of the usable surface. The drawing sheet numbering must be 
clear and larger than the numbers used as reference characters to avoid confusion. The number of each sheet should be 
shown by two Arabic numerals placed on either side of an oblique line, with the first being the sheet number and the 
second being the total number of sheets of drawings, with no other marking. 
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6. The drawings are objected to under 37 CFR 1 .84(t) for failing to properly number the 
sheets of drawings - it is suggested to either remove the drawing sheet numbers or place them in 
the middle of the top of the sheet, but not in the margin. 

7. The following is a quote from 37 CFR 1 .84(h): 

All views of the drawing must be grouped together and arranged on the sheet(s) without wasting space, 
preferably in an upright position, clearly separated from one another, and must not be included in the sheets 
containing the specifications, claims, or abstract. 

8. FIG. 3 - FIG. 1 1 are objected to under 37 CFR 1 .84(h) for failing to group together and 
arrange on the sheets without wasting space - it is suggest to group the figures in such a way to 
not waste space (e.g. FIG. 3, FIG. 4 grouped on one sheet). 

Specification 

8. Applicant is reminded of the proper language and format for an abstract of the disclosure 
as it has been noted the abstract is 235 words in length and greater than 150. 

The abstract should be in narrative form and generally limited to a single paragraph on a 
separate sheet within the range of 50 to 150 words. It is important that the abstract not exceed 
150 words in length since the space provided for the abstract on the computer tape used by the 
printer is limited. The form and legal phraseology often used in patent claims, such as "means" 
and "said," should be avoided. The abstract should describe the disclosure sufficiently to assist 
readers in deciding whether there is a need for consulting the full patent text for details. 

The language should be clear and concise and should not repeat information given in the 
title. It should avoid using phrases which can be implied, such as, "The disclosure concerns," 
"The disclosure defined by this invention," "The disclosure describes," etc. 
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9. The title of the invention is not descriptive. A new title is required that is clearly % 
indicative of the invention to which the claims are directed. 



Claim Objections 

1 0. The following is a quotation of 37 CFR 1 .75(a): 

The specification must conclude with a claim particularly pointing out and distinctly claiming the subject 
matter which the applicant regards as his invention or discovery. 

1 1 . Claims 1 - 10 are objected to under 37 CFR 1 .75(a), as failing to conform to particularly 
point out and distinctly claim the subject matter which application regards as his invention or 
discovery. 

(i) Claim 1, line 10; claim 10, line 1 1 appears to have a grammatical error - suggest 
changing to ". . .information containing image of re s p e ctiv e of the obstacle. . ." 

(ii) Claim 1 , line 1 1 ; claim 10, line 12 cites ", and like images;" but it is unclear what the 
like image are - suggest removing the indefinite limitation altogether as assumed for 
examination purposes 

12. The following is a quotation of 37 CFR 1.75(c): 

One or more claims may be presented in dependent form, referring back to and further limiting another 
claim or claims in the same application. Any dependent claim which refers to more than one other claim 
("multiple dependent claim") shall refer to such other claims in the alternative only. A multiple dependent 
claim shall not serve as a basis for any other multiple dependent claim. For fee calculation purposes under § 
1 . 16, a multiple dependent claim will be considered to be that number of claims to which direct reference is 
made therein. For fee calculation purposes also, any claim depending from a multiple dependent claim will 
be considered to be that number of claims to which direct reference is made in that multiple dependent 
claim. In addition to the other filing fees, any original application which is filed with, or is amended to 
include, multiple dependent claims must have paid therein the fee set forth in § 1.1 6(j). Claims in 
dependent form shall be construed to include all the limitations of the claim incorporated by reference into 
the dependent claim. A multiple dependent claim shall be construed to incorporate by reference all the 
limitations of each of the particular claims in relation to which it is being considered. 



• t 

Application/Control Number: 10/522,236 Page 5 

Art Unit: 2624 

13. Claim 5 is objected to under 37 CFR 1.75(c) as being in improper form because a 
multiple dependent claim cannot depend from any other multiple dependent claim. See MPEP 
§ 608.01 (n). Accordingly, the claim has not been further treated on the merits. 

14. Claim 9 is objected to under 37 CFR 1 .75(c), as being of improper dependent form for 
failing to further limit the subject matter of a previous claim. Applicant is required to cancel the 
claim, or amend the claim to place the claim in proper dependent form, or rewrite the claim in 
independent form. It is already known that any faculty information, player's position 
information, instrument information, and rule information are based on knowledge about a sport 
as a subject for image extraction. If otherwise, the rules of the sport would be "made up" and 
changeable. 

Claim Rejections - 35 USC §101 

15. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

16. The USPTO "Interim Guidelines for Examination of Patent Applications for Patent 
Subject Matter Eligibility" (Official Gazette notice of 22 November 2005), Annex IV, reads as 
follows: 

Descriptive material can be characterized as either "functional descriptive material" or "nonfunctional descriptive 
material." In this context, "functional descriptive material" consists of data structures and computer programs 
which impart functionality when employed as a computer component. (The definition of "data structure" is "a 
physical or logical relationship among data elements, designed to support specific data manipulation functions." 
The New IEEE Standard Dictionary of Electrical and Electronics Terms 308 (5th ed. 1993).) "Nonfunctional 
. descriptive material" includes but is not limited to music, literary works and a compilation or mere arrangement 
of data. 

When functional descriptive material is recorded on some computer-readable medium it becomes structurally and 
functionally interrelated to the medium and will be statutory in most cases since use of technology permits the 



f 

Application/Control Number: 10/522,236 Page 6 

Art Unit: 2624 

function of the descriptive material to be realized. Compare In re Lowry, 32 F.3d 1579, 1583-84, 32 USPQ2d 
1031, 1035 (Fed. Cir. 1994) (claim to data structure stored on a computer readable medium that increases 
computer efficiency held statutory) and Warmerdam, 33 F.3d at 1360-61, 31 USPQ2d at 1759 (claim to 
computer having a specific data structure stored in memory held statutory product-by-process claim) with 
Warmerdam, 33 F.3d at 1361, 31 USPQ2d at 1760 (claim to a data structure per se held nonstatutory). 

In contrast, a claimed computer-readable medium encoded with a computer program is a computer element 
which defines structural and functional interrelationships between the computer program and the rest of the 
computer which permit the computer program's functionality to be realized, and is thus statutory. See Lowry, 32 
F.3d at 1583-84, 32 USPQ2d at 1035. 

17. Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to 
non-statutory subject matter as follows. Claim 10 defines "[a]n image recognition program with 
a computer" embodying functional descriptive material. However, the claim does not define a 
computer-readable medium or memory and is thus non-statutory for that reason (i.e., "When 

t 

functional descriptive material is recorded on some computer-readable medium it becomes 
structurally and functionally interrelated to the medium and will be statutory in most cases since 
use of technology permits the function of the descriptive material to be realized" - Guidelines 
Annex IV). That is, the scope of the presently claimed "image recognition program with a 
computer" can range from paper on which the program is written, to a program simply 
contemplated and memorized by a person. The examiner suggests amending the claim to 
embody the program on "computer-readable medium" or equivalent in order to make the claim 
statutory ( e.g. "A computer readable medium encoded with an image recognition program such 
that when executed, causes an image recognition apparatus to operate..." ). Any amendment to 
the claim should be commensurate with its corresponding disclosure. 

Claim Rejections - 35 USC § 103 

18. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 
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(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

19. Claims 1 - 2 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over 

• th 

Pingali et al. ( Ball Tracking and Virtual Replays for Innovative Tennis Broadcasts , 15 
International Conference on Pattern Recognition, 2000, Proceedings, Vol. 4, pg 152 - 156) in 
view of Pizano et al. (US 6,101,274 A). 

Regarding claim 1, while Pingali discloses an image recognition apparatus (FIG. 1; 
Section 2.1, pp 152 - 153) for recognizing movements of players matched against each other 
("the players" in Introduction, p 152) between domains partitioned with such an obstacle as net 
in a sport match or game (tennis net in FIG. 2; domains partitioned in FIG. 4) from contents 
including a television program being telecasted to show the sport match or game (FIG. 2; 
"broadcasting" in Introduction, p 152), an image material in an uncompleted state for 
broadcasting and contents recorded in such a recording medium as a VTR (Introduction, p 152; 
FIG. 2), the image recognition apparatus comprising: 

a score information obtaining section (the section responsible for keeping track of a 
"changing game score", Section 3.1, p 154) configured to obtain score information indicative of 
scores of the respective players which vary as the sport match or game proceeds; 

a play event information obtaining section (FIG. 1 ; "Trigger received?" in FIG. 1) 
configured to obtain play event information indicative of a characteristic movement of each of 
the players from picture information included in the contents (the frames are captured upon the 
serve/hit of the tennis ball that causes a projection, and since each player serves/hits, the 
characteristic movement of each of the players from the picture information is the serve/hit 
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itself), the picture information containing images of respective of the obstacle and the players 
(FIG. 2), and like images (boundaries of the tennis court in FIG. 2); and 

an image substance recognizing section (the section responsible for "score-based 
queries", Section 3.1, p 154; the section responsible for keeping track of a "changing game 
score", Section 3.1, p 154) configured to make a comparison between a score information item 
obtained immediately before a point in time of generation of the play event information and a 
score information item obtained immediately after the point in time and make reference to a 
result brought by the play event information (FIG. 2 shows an example of tracking the ball from 
a suggested unreturned hit; an unreturned hit gives a new score to one of the players; the tracked 
path/projection of the ball immediately before and after hitting the ground in an out-of-bound 
area would be an automatic change in score, thus "a score information item" being where the 
tennis ball hit the ground), thereby recognizing a substance of an image provided by the play 
event information (this would recognize a hit/serve by the other player from the one who did not 
return the ball - both actions on behalf of each player are "play event information"), Pingali does 
not teach wherein the score information itself is displayed on a screen. 

Pizano discloses an apparatus for detecting and interpreting texual captions in digital 
video signals (FIG. 6) that teaches wherein the score information itself is displayed on a screen 
("3) SPORTS/TRANSPARENT" in FIG. 1; the score in FIG. 5). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the screen of Pingali to include score information as taught by Pizano"to identify 
the collection of video frames that contain text captions.", Pizano, Col. 5, lines 54 - 56 and "to 
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create a score browser which would enable a person to move directly to specific portions of the 
video", Pizano, Col. 1, lines 61 - 63. 

Regarding claim 2, while Pingali in view of Pizano disclose the image recognition 
apparatus according to claim 1, Pingali in view of Pizano do not teach wherein the score 
information obtaining section is configured to obtain the score information from at least one of 
the picture information included in the contents, sound information including commentary voice 
of a commentator, and data information transmitted as multiplexed on radio waves during 
broadcasting. 

Pizano discloses an apparatus for detecting and interpreting texual captions in digital 
video signals (FIG. 6; FIG. 7; FIG. 10) that teaches wherein a score information obtaining 
section (section responsible for FIG. 6, FIG. 7, FIG. 10) is configured to obtain the score 
information (bottom pictures of the score in FIG. 10) from at least one of the picture information 
(top picture in FIG. 10) included in the contents (the contents being everything displayed at the 
top picture in FIG. 10), sound information including commentary voice of a commentator, and 
data information transmitted as multiplexed on radio waves during broadcasting. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the score information obtaining section of Pingali in view of Pizano to include 
configuring to obtain the score information from at least one of the picture information included 
in the contents, sound information including commentary voice of a commentator, and data 
information transmitted as multiplexed on radio waves during broadcasting as taught by 
Pizano"to identify the collection of video frames that contain text captions.", Pizano, Col. 5, 
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lines 54-56 and "to create a score browser which would enable a person to move directly to 
specific portions of the video", Pizano, Col. 1, lines 61 - 63. 

Regarding claim 10, claim 1 recites identical features as in claim 10. Thus, 
references/arguments equivalent to those presented above for claim 1 are equally applicable to 
claim 10. 

20. Claims 3 - 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Pingali et al. 
( Ball Tracking and Virtual Replays for Innovative Tennis Broadcasts , 15 International 
Conference on Pattern Recognition, 2000, Proceedings, Vol. 4, pg 152 - 156) in view of Pizano 
et al. (US 6,101,274 A) and Sudhir et al. ( Automatic Classification of Tennis Video for High- 
level Content-based Retrieval Proceedings of the 1998 International Workshop on Content- 
Based Access of Image and Video Databases (CAIVD '98), 1998, pp 81 - 90). 

Regarding claim 3, while Pingali in view of Pizano disclose image recognition apparatus 
according to claims 1 or 2, further comprising: 

a domain element extracting section (section responsible for obtaining all coordinates to 
track/detect/match instruments, obstacle, and boundaries of the tennis court as shown in FIG. 2 
and FIG. 4) configured to extract instrument information on an instrument moving between the 
domains to serve as an object of score count in the sport match or game (an ace by Sampras in 
FIG. 3 was within the tennis boundary domain, thus a scoring count) 

Pingali does not teach a domain element extracting section configured to extract from the 
picture information facility information including information on the obstacle, information on the 
domains and information on boundary lines between the domains and an area outside the 
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domains, and player's position information indicative of a player's position; rule information 
storage section configured to store rule information on the sport match or game; and basic 
movement storage section configured to store basic movement information on players' 
characteristic movements generalized in a sport of concern, wherein the play event information 
obtaining section includes a play event information determining section configured to determine 
a play event information item on a play event characteristic of each of the players included in the 
picture information as the play event information based on domain elements extracted from the 
picture information, the rule information, and the basic movement information stored in the basic 
movement information storage section. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, pg 81) that teaches 

a domain element extracting section (FIG. 3; Table 2) configured to extract from the 
picture information facility information including information on the obstacle (dashed line in 
FIG. 7), information on the domains and information on boundary lines between the domains and 
an area outside the domains (FIG. 7), and player's position information indicative of a player's 
position ("Player Tracking Module" in FIG. 1); 

rule information storage section ("court-line detection module" in FIG. 1 for storing rules 
on court-line boundaries) configured to store rule information on the sport match or game; and 

basic movement storage section ("Player Tracking Module" in FIG. 1) configured, to 
store basic movement information on players' characteristic movements generalized in a sport of 
concern, wherein a play event information obtaining section ("High-level Reasoning module" in 
deciding plays like "high-level events like baseline-rallies, passing-shots, serve-and-volleying 
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and net-games" in Section 9, p 89) includes a play event information determining section 
configured to determine a play event information item on a play event characteristic of each of 
the players included in the picture information as the play event information based on domain 
elements (the boxes outlining the players on the right side of FIG. 6 are the domains) extracted 
from the picture information, the rule information (FIG. 7), and the basic movement information 
stored in the basic movement information storage section (FIG. 1). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano to include a domain element extracting 
section configured to extract from the picture information facility information including - 
information on the obstacle, information on the domains and information on boundary lines 
between the domains and an area outside the domains, and player's position information 
indicative of a player's position; rule information storage section configured to store rule 
information on the sport match or game; and basic movement storage section configured to store 
basic movement information on players' characteristic movements generalized in a sport of 
concern, wherein the play event information obtaining section includes a play event information 
determining section configured to determine a play event information item on a play event 
characteristic of each of the players included in the picture information as the play event 
information based on domain elements extracted from the picture information, the rule 
information, and the basic movement information stored in the basic movement information 
storage section as taught by Sudhir as "there is dire need for algorithms that are able to 
automatically infer high-level content from data.", Sudhir, Introduction, p 81. 
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Regarding claim 4, while Pingali in view of Pizano and Sudhir disclose the apparatus 
according to claim 3, Pingali in view of Pizano and Sudhir do not teach wherein the player's 
position information is position information indicative of a domain containing each of the 
players and the instrument constantly held and used by the player. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, pg 81) that teaches wherein the player's position information is position 
information indicative of a domain containing each of the players and the instrument constantly 
held and used by the player (the player, the tennis racquet, and tennis ball at the point of impact 
are all in the boxes (domain) on the right side of FIG. 6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano and Sudhir to include wherein the 
player's position information is position information indicative of a domain containing each of 

* 

the players and the instrument constantly held and used by the player as taught by Sudhir as 
"there is dire need for algorithms that are able to automatically infer high-level content from 
data.", Sudhir, Introduction, p 8 1 . 

Regarding claim 5, Pingali discloses wherein the domain element extracting section 
(section responsible for obtaining all coordinates to track/detect/match instruments, obstacle, and 
boundaries of the tennis court as shown in FIG. 2 and FIG. 4) is configured to extract the 

» 

player's position information (the origination of the path of the tennis ball in FIG. 2 and FIG. 4 is 

* 

the player's position information) from the picture information (FIG. 4) based on the facility 
information (coordinates to track/detect/match instruments, obstacle, and boundaries in FIG. 4 
and FIG. 2) extracted by the domain element extracting section. 
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Regarding claim 6, Pingali discloses wherein the domain element extracting section 
(section responsible for obtaining all coordinates to track/detect/match instruments, obstacle, and 
boundaries of the tennis court as shown in FIG. 2 and FIG. 4) is configured to extract the 
instrument information from the picture information (FIG. 2, FIG. 4, path of the tennis ball) 
based on the facility information (coordinates to track/detect/match instruments, obstacle, and 
boundaries in FIG. 4 and FIG. 2) and the player's position information (the origination of the 
path of the tennis ball in FIG. 2 and FIG. 4 is the player's position information) extracted by the 
domain element extracting section. 

Regarding claim 7, while Pingali in view of Pizano and Sudhir disclose the apparatus 
according to claim 3, Pingali in view of Pizano and Sudhir do not teach wherein the play event 
information obtaining section includes a play event index information output section configured 
to output plural play event information items determined by the play event information 
determining section as arranged in a time sequence. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, pg 81) that teaches wherein the play event information obtaining section 
includes a play event index information output section (Table 3, p 88) configured to output plural 
play event information items ("High-level Annotation" column in Table 3) determined by the 
play event information determining section as arranged in a time sequence (the table suggests the 
arrangement of time sequence). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano and Sudhir to include wherein the play 
event information obtaining section includes a play event index information output section 
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configured to output plural play event information items determined by the play event 
information determining section as arranged in a time sequence as taught by Sudhir as "there is 
dire need for algorithms that are able to automatically infer high-level content from data.", 
Sudhir, Introduction, p 81. 

Regarding claim 8, while Pingali in view of Pizano and Sudhir disclose the apparatus 
according to claim 3, Pingali in view of Pizano and Sudhir do not teach wherein the play event 
index information output section is configured to output the play event information items 
together with instrument information items in a time sequence. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, pg 81) that teaches wherein the play event index information output 
section (Table 3, p 88) is configured to output the play event information items ("High-level 
Annotation" column in Table 3) together with instrument information items ("BL" is "Baseline", 
and thus the table indicates the location information from which the instrument was during the 
serve) in a time sequence (the table suggests the arrangement of time sequence). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano and Sudhir to include wherein the play 
event index information output section is configured to output the play event information items 
together with instrument information items in a time sequence as taught by Sudhir as "there is 
dire need for algorithms that are able to automatically infer high-level content from data.", 
Sudhir, Introduction, p 8 1 . 
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Regarding claim 9, Pingali discloses wherein the facility information, the player's 
position information, the instrument information and the rule information are based on 
knowledge about a sport as a subject for image extraction (Introduction, p 152). 

Conclusion 

2 1 . The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure: US 5923365 A; US 6031568 A; US 6071002 A; US 6072504 A; US 6101274 A; US 
6231443 Bl; US 20020018594 Al; US 20020159637 Al. 

22. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to David P. Rashid whose telephone number is (571) 270-1578. 
The examiner can normally be reached Monday - Friday 8:30 - 17:00 ET. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Vikkram Bali can be reached on (571) 272-7415. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 

# 

applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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